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SUMMARY  PAGE 


Problem:  Experts  in  many  domains  possess  and  utilize  mental  models  that  novice  personnel 
have  not  yet  developed.  As  people  gain  more  experience,  their  mental  models  tend  to  become 
more  homogeneous  with  other  experts,  compared  to  the  novice  people.  Towards  this  end, 
system  developers  and  training  researchers  attempt  to  identify  critical  components  of  expert 
operator  assessment  and  knowledge.  Via  analyses  of  submariner  knowledge  for  concepts  related 
to  responsibilities  for  the  Officer  of  the  Deck  (OOD)  task,  this  study  examined  how  training  may 
alter  knowledge  representation  and  priority  of  conceptual  importance  and  how  overlap  in  mental 
models  may  be  due  to  amount  of  experience. 

Method:  Eighty-three  Naval  Officers  performed  a  card  sorting  and  ranking  task  to  indicate 
similarity  and  relative  importance  among  20  categories  of  information  available  on  submarines 
in  different  operational  environments.  These  submarine  concepts  consisted  of  types  of 
information  the  submarine  OOD  would  encounter  while  on  watch,  with  the  exception  of  tactics 
information.  Participants  were  instructed  to  sort  the  cards  into  piles  according  to  similarity  and 
were  not  constrained  by  the  number  of  piles  they  could  create.  Following  the  sorting  task 
participants  ranked  the  cards  according  to  relative  importance  for  four  scenarios.  They  were 
instructed  to  create  a  single  pile  of  cards  ordered  from  the  most  important  piece  of  submarine 
information  to  the  least  important.  Data  were  collected  from  Submarine  Officer  Basic  Course 
students  pre  and  post  course,  Submarine  Officer  Advanced  Course  students,  and  Post 
Department  Head  submariners. 

Findings:  In  support  of  shared  mental  model  theory  and  consistent  with  other  studies  of 
knowledge  organization  and  the  mental  models  of  submariners,  this  investigation  found  that 
more  experienced  personnel  view  conceptual  linkages  more  similarly  and  showed  higher 
agreement  with  an  expert  model  (Fiore,  Fowlkes,  Martin-Milham,  &  Oser,  2000).  Conversely, 
the  least  experienced  personnel  see  conceptual  linkages  less  similarly,  but,  following  training, 
these  participants  were  found  to  view  the  conceptual  linkages  more  similarly  to  the  experienced 
personnel.  From  the  standpoint  of  understanding  the  importance  of  particular  concepts, 
significant  differences  between  experienced  and  novice  personnel  were  identified  prior  to,  and 
following,  training.  These  data  highlight  how  participants  vary  in  their  understanding  of  the 
importance  of  these  concepts. 

Application:  Knowledge  of  the  mental  models  that  experienced  submarine  officers  possess  can 
provide  input  into  training  objectives  for  less  experienced  personnel.  Moreover,  knowledge  of 
how  experienced  submariners  place  priority  on  certain  information  items  in  different  situations 
can  guide  training  for  submariners  early  in  their  career. 

ADMINISTRATIVE  INFORMATION 

This  work  was  conducted  under  Work  Unit  51001  entitled:  "Information  Requirements  and  Information 
Organizations  in  Submarine  Combat  Systems."  The  opinions  or  assertions  contained  herein  are  the  private  ones  of 
the  authors  and  are  not  to  be  construed  as  official  or  reflecting  the  views  of  the  Department  of  the  Navy,  Department 
of  Defense,  of  the  United  States  Government.  This  research  has  been  conducted  in  compliance  with  all  applicable 
Federal  Regulations  governing  the  Protection  of  Human  Subjects  in  Research.  This  Technical  Report  was  approved 
for  publication  on  12  July  2004,  and  designated  as  NSMRL  Technical  Report  TR  #1235. 
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ABSTRACT 


It  is  increasingly  being  recognized  that  understanding  expert  knowledge  structures  associated 
with  critical  decision  processes  may  facilitate  Naval  personnel  performance.  Towards  this  end, 
system  developers  and  training  researchers  attempt  to  identify  critical  components  of  expert 
operator  assessment  and  knowledge.  Differing  domains  of  practice  rely  to  varying  degrees  on 
perceptual  and  conceptual  knowledge.  Perceptual  knowledge  is  relied  upon  for  recognizing 
critical  cues  in  the  environment  whereas  conceptual  knowledge  is  used  to  interpret  the  meaning 
and  importance  of  these  cues.  Via  analyses  of  submariner  knowledge  for  concepts  related  to 
responsibilities  for  the  Officer  of  the  Deck  (OOD)  task,  this  study  examined  how  training  may 
alter  knowledge  representation  and  priority  of  conceptual  importance  and  how  overlap  in  mental 
models  may  be  due  to  amount  of  experience. 

Eighty-three  Naval  Officers  performed  a  card  sorting  and  ranking  task  to  indicate  similarity  and 
relative  importance  among  20  categories  of  information  available  on  submarines  in  different 
operational  environments.  These  submarine  concepts  consisted  of  types  of  information  the 
submarine  OOD  would  encounter  while  on  watch,  with  the  exception  of  tactics  information. 
Participants  were  instructed  to  sort  the  cards  into  piles  according  to  similarity  and  were  not 
constrained  by  the  number  of  piles  they  could  create.  Following  the  sorting  task  participants 
ranked  the  cards  according  to  relative  importance  for  four  scenarios.  They  were  instructed  to 
create  a  single  pile  of  cards  ordered  from  the  most  important  piece  of  submarine  information  to 
the  least  important.  Data  were  collected  from  Submarine  Officer  Basic  Course  (SOBC)  students 
pre  and  post  course,  Submarine  Officer  Advanced  Course  (SOAC)  students,  and  Post 
Department  Head  submariners  (PostDH). 

Correlations  between  and  among  participant  groups  (SOBCpre,  SOBCpst,  SOAC,  PostDH) 
revealed  differences  in  cognitive  organization  of  the  submarine  OOD  concepts.  Within  group 
comparisons  showed  that  the  degree  of  mental  model  similarity  was  significantly  greater  for 
more  experienced  participants.  When  considering  across  group  comparisons,  analyses  showed 
that  the  degree  of  across  group  mental  model  similarity  was  greatest  for  SOAC  and  SOBCpst. 
Moreover,  SOAC  students  were  in  higher  agreement  with  the  expert  composite  model  generated 
from  Post  DH  submariners,  and  concept  rankings  varied  as  a  function  of  experience. 

In  support  of  shared  mental  model  theory  and  consistent  with  other  studies  of  knowledge 
organization  and  the  mental  models  of  submariners,  this  investigation  found  that  more 
experienced  personnel  view  conceptual  linkages  more  similarly  and  showed  higher  agreement 
with  an  expert  model  (Fiore,  Fowlkes,  Martin-Milham,  &  Oser,  2000).  Conversely,  the  least 
experienced  personnel  see  conceptual  linkages  less  similarly,  but,  following  training,  these 
participants  were  found  to  view  the  conceptual  linkages  more  similarly  to  the  experienced 
personnel.  From  the  standpoint  of  understanding  the  importance  of  particular  concepts, 
significant  differences  between  experienced  and  novice  personnel  were  identified  prior  to,  and 
following,  training.  These  data  highlight  how  participants  vary  in  their  understanding  of  the 
importance  of  these  concepts. 
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INTRODUCTION 


It  is  increasingly  being  recognized  that  understanding  expert  knowledge  structures  associated 
with  critical  decision  processes  may  facilitate  Naval  personnel  performance  (e.g.,  Gray  & 
Kirschenbaum,  2000;  Kirschenbaum,  2001).  Towards  this  end,  system  developers  and  training 
researchers  attempt  to  identify  critical  components  of  expert  operator  assessment  and  decision 
processes.  From  the  standpoint  of  system  development,  such  findings  may  aid  in  the 
identification  of  important  parameters  to  be  included  in  relevant  operational  environments  (e.g., 
Schvaneveldt,  Beringer,  &  Lamonica,  2001).  From  the  standpoint  of  training  and  performance, 
targets  for  learning  may  be  identified  (e.g.,  Fiore,  Fowlkes,  Martin-Milham,  &  Oser,  2000).  As 
such,  an  understanding  of  the  knowledge  structures  associated  with  expert  decision  making  is 
critical  to  the  development  of,  not  only  cognitively  valid  systems,  but  also  efficient  training 
programs. 

Performance  in  many  of  today's  complex  tasks  (e.g.,  aviation,  nuclear  power  plants)  requires  that 
operators  have  the  ability  to,  not  only  rapidly  recognize  cues  in  the  environment,  but  also  to 
interpret  the  meaning  and  importance  of  these  cues.  This  former  ability  is  described  as  a  process 
involving  perceptual  knowledge,  defined  here  as  the  veridical  and  largely  unverbalizable 
representation  that  develops  via  multiple  exposures  to  environmental  stimuli.  The  latter  ability  is 
described  as  a  process  involving  conceptual  knowledge,  defined  as  a  relatively  verbalizable 
component  of  knowledge  which  is  developed  through  the  integration  and  structuring  of 
information  into  meaningful  representations  (e.g.,  Lesgold,  Rubinson,  Feltovich,  Glaser,  Klopfer, 
&  Wang,  1988;  Melcher  &  Schooler,  in  press).  Previous  research  has  investigated  the  nature  of 
expert  cognitive  processes  involved  in  pilot  instrument  scans  within  this  distinction  of  perceptual 
and  conceptual  knowledge  (Fiore,  Jentsch,  Oser,  &  Cannon-Bowers,  2000),  suggesting  that  such 
tasks  rely  on  knowledge  types  relevant  to  expert  cognitive  processes  in  a  variety  of  domains. 
Understanding  how  this  knowledge  is  represented,  as  well  as  the  processes  with  which  decisions 
are  made,  is  critical  to  cognitive  engineering  and  decision  making  research.  For  example,  such 
an  understanding  may  facilitate  the  development  of  accelerated  training  programs  or  improved 
system  design.  Lesgold  et  al.  (1988)  noted  that  task  novices  initially  develop  perceptual 
expertise  more  rapidly  than  conceptual  expertise,  but,  with  experience,  the  conceptual  expertise 
gains  strength. 

For  the  following  reasons  this  investigation  focused  on  conceptual  knowledge  given  its 
importance  in  the  submariner  environment.  First,  the  environment  in  which  a  submarine's  crew 
operates  differs  from  the  normal  environment  in  significant  ways.  The  submarine  environment  is 
a  fluid-filled  medium,  little  or  no  direct  visual  information  is  provided,  and  much  of  the  acoustic 
information  is  made  available  only  indirectly  and  that  too  is  in  visual  form.  As  a  consequence, 
mental  representations  of  the  submariner's  world  can  be  greatly  impacted.  Second,  through 
technological  advances,  increasing  amounts  of  information  are  available  to  the  submarine 
watchstander,  especially  the  Officer  of  the  Deck  (OOD).  The  OOD  controls  the  boat  and 
executes  his  mission  by  making  sense  of  the  substantial  amount  of  information  presented  by 
external  sensors  and  internal  processing.  The  OOD's  task  essentially  is  ship's  safety  and 
navigation.  The  OOD  is  strategically  located  at  the  control  center,  and  receives  a  variety  of 
diverse  information  as  inputs,  making  operational  decisions  based  on  his  understanding.  The 
OOD  uses  this  information  to  build  his  mental  model,  which  is  important  for  developing  and 
maintaining  situational  awareness.  Too  much  or  irrelevant  information  given  to  the  OOD  can 
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degrade  performance  of  the  man-machine  system  rather  than  enhance  it.  The  amount  of 
information  available  greatly  exceeds  the  capacity  of  human  working  memory,  so  they  manage 
this  information  load  by  cognitively  organizing  it  in  such  a  way  that  needed  information  can  be 
retrieved  from  memory  as  accurately  and  quickly  as  possible.  The  current  research  describes  and 
models  the  organization  of  submarine  information  used  by  qualified  OODs.  Additionally,  the 
development  of  these  cognitive  models  from  novices  to  experts  is  explored. 

A  variety  of  quantitative  and  qualitative  methods  can  be  used  to  measure  knowledge  structures, 
each  possessing  unique  advantages  and  disadvantages.  Fiore,  Fowlkes,  Martin-Milham,  and 
Oser  (2000)  found  a  degree  of  convergence  in  knowledge  structure  assessment  using  similarity 
ratings  (e.g.,  Schvaneveldt,  1990)  and  card  sorting  techniques.  Specifically,  card  sorting  and 
similarity  ratings  were  used  to  assess  the  manner  in  which  experts  view  conceptual  relations 
associated  with  situation  awareness.  This  study  showed  that  knowledge  structures  measured 
with  card  sorting  were  significantly  related  to  knowledge  structures  elicited  using  more  time- 
consuming  methods  such  as  Pathfinder  similarity  ratings.  Further,  although  card  sorts  are  a 
somewhat  limited  method  because  trainees  are  forced  to  group  together  items  rather  rigidly, 
previous  research  does  document  a  relation  between  task  performance  and  accurate  mental 
model  development  measured  via  card  sorts.  For  example,  research  with  card  sorting  has 
documented  how  training  systems  that  enable  learners  to  build  an  appropriate  mental  model  of 
the  relations  between  concepts  can  encourage  the  acquisition  of  knowledge  structures  more 
similar  to  an  expert  model  (e.g.,  Fiore,  Cuevas,  &  Oser,  2003)  and  also  how  cognitively 
diagnostic  assessment  for  complex  task  training  can  proceed  (Cuevas,  Fiore,  &  Oser,  2002; 

Fiore,  Cuevas,  Scielzo,  &  Salas,  2002).  In  short,  card  sorting  was  chosen  because  it  has  been 
shown  to  be  an  effective  tool  in  identifying  the  level  of  organization  of  core  concepts  as  well  as 
being  a  reliable  indicator  of  how  novices  evaluate  concepts  (Fiore  et  al.,  2002;  2003),  and  experts 
view  conceptual  relations  (Fiore,  Jentsch,  Oser,  &  Cannon-Bowers,  2000). 

OBJECTIVES 

Two  approaches  were  used  to  converge  on  an  understanding  of  how  knowledge  related  to 
complex  tasks  may  change  with  experience.  This  investigation  examined  how  training  may  alter 
knowledge  representation,  and  how  overlap  in  mental  models  may  be  due  to  amount  of 
experience.  Similar  research  has  assessed  the  nature  and  sharedness  of  mental  models  for 
submariners.  For  example,  Smith- Jentsch,  Campbell,  Milanovich,  &  Reynolds  (2001)  looked  at 
how  experience  influenced  mental  models  of  teamwork  and  found  greater  agreement  based  upon 
either  rank  or  years  in  service.  Additionally,  when  compared  to  an  expert  model,  those  with  a 
higher  rank  showed  greater  agreement.  The  current  research  expands  upon  that  work  to  examine 
how  mental  models  associated  with  the  OOD  task  may  show  similar  patterns  and  further 
investigate  how  information  priority  may  change  due  to  experience.  Priority  rankings  data  for 
task  related  concepts  may  elucidate  differences  in  conceptual  knowledge  across  groups  varying 
in  experience.  For  example,  in  a  study  looking  at  phases  of  flight,  Schvaneveldt,  Beringer,  & 
Lamonica  (2001)  found  that  this  technique  was  able  to  highlight  differences  in  perceptions  of 
importance  dependent  upon  the  level  of  expertise  of  the  pilots. 

Based  upon  shared  mental  model  theory  (e.g.,  Cannon-Bowers,  Salas,  &  Converse,  1993;  Fiore, 
Salas,  &  Cannon-Bowers,  2001),  it  was  hypothesized:  (1)  that  experienced  participants  would 


show  greater  similarity  within  their  groups  and  to  an  expert  model;  (2)  following  training,  novice 
participants  would  show  greater  agreement  in  mental  models  for  the  critical  concepts,  and  show 
greater  agreement  with  the  experienced  participants.  The  present  research  additionally  examined 
the  degree  to  which  these  participants'  mental  models  were  similar  to  an  expert  model  (i.e.,  a 
composite  model  based  upon  the  input  of  a  group  of  expert  participants).  Additionally,  rankings 
of  importance  for  these  critical  concepts  were  expected  to  show  how  novice  and  less  experienced 
personnel  may  misunderstand  priorities  associated  with  decision-making  criteria  when  compared 
to  experts.  Thus,  the  overall  goal  was  to  use  differing  knowledge  elicitation  techniques  in  order 
to  develop  an  understanding  of  how  it  is  that  experts  and  novices  both  classify  the  concepts 
associated  with  complex  decision  making  tasks  and  understand  the  importance  of  these  concepts 
(e.g.,  Fiore,  Fowlkes,  Martin-Milham,  &  Oser,  2000). 

METHODS 


Participants 

Participants  were  drawn  from  the  submarine  community  to  represent  novice,  experienced,  and 
expert  groups.  Novice  participants  (. N  =  39)  were  students  in  their  first  week  of  Submarine 
Officer  Basic  Class  (SOBCpre)  and  last  week  of  class  (SOBCpst)  with  no  OOD  experience.  The 
SOBCpre  and  SOBCpst  groups  consisted  of  the  same  subjects.  The  SOBCpst  group  was  not 
OOD-qualified,  but  they  had  all  the  relevant  lectures  and  trainer  experience.  Experienced 
participants  (. N  =  40)  were  qualified  OODs  in  Submarine  Officer  Advanced  Class  (SOAC).  As 
shown  in  Table  1,  Years  in  Service  for  participants  in  the  SOBC  group  ( M  =  5.12  years)  was 
significantly  lower  than  that  of  participants  in  the  SOAC  group  (M  =  10.58  years),  F(l,  89)  = 
34.76,  p  <  .001.  Years  in  Present  Rank  for  participants  in  the  SOBC  group  (M  =1.6  years)  was 
significantly  lower  than  that  of  participants  in  the  SOAC  group  ( M  =  3.3  years),  F(  1 , 88)  = 

62.66,  p  <  .001.  Years  Sea  Duty  for  participants  in  the  SOBC  group  (M  =1.1  years)  was 
significantly  lower  than  that  of  participants  in  the  SOAC  group  (M  =3.8  years),  F(\,  89)  = 

35.92,  p  <  .001.  Data  from  a  small  group  of  experts  (. N  =  4)  were  also  collected  in  order  to 
develop  a  composite  model  for  comparison  to  less  experienced  participants.  The  expert 
participants  were  all  Lieutenant  Commanders  (04)  and  overall,  had  an  average  length  of  8.9 
years  in  that  rank.  The  mean  number  of  years  in  service  was  20.4,  with  an  average  of  6.5  years 
submarine  sea  duty.  All  of  the  expert  participants  had  served  as  department  heads  on  submarines 
and  were  currently  filling  a  Post-Department  Head  shore  tour  billet.  Table  2  provides 
information  on  the  ranks  of  the  different  subject  groups. 
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Table  1.  Subjects' Demographic  Information 


Characteristic 

N 

Mean 

SD 

Age 

SOBC 

39 

25.2 

2.4 

SO  AC 

40 

31.2 

2.4 

Post  DH 

4 

40.5 

0.6 

Years  of  Service 

SOBC 

39 

5.1 

3.6 

SO  AC 

40 

10.6 

3.5 

Post  DH 

4 

20.4 

1.9 

Years  of  Sea  Duty 

SOBC 

39 

1.1 

1.3 

SO  AC 

40 

3.8 

1.2 

Post  DH 

4 

6.5 

0.6 

Table  2.  Rank  Distribution  of  Subjects 


Rank 

01 

02 

03 

04 

05 

Group 

SOBC 

37 

2 

0 

0 

0 

SO  AC 

0 

0 

39 

1 

0 

Post  DH 

0 

0 

0 

4 

0 

Materials 

Various  types  of  information  from  current  and  proposed  submarine  systems  were  classified  into 
20  categories.  These  20  categories  are  listed  in  Appendix  A.  Descriptions  of  these  categories 
comprised  the  20  stimuli  for  the  tasks  to  be  performed  by  the  subjects.  Each  of  the  stimuli  was 
typed  on  a  separate  3"  X  5"  card  and  numbered  on  the  reverse  side  to  create  the  stimulus  deck. 
Some  information  available  to  the  OOD  (e.g.,  torpedo  load  out,  weapons  safeties, 
countermeasures  status,  intelligence  on  contacts  in  the  area)  was  not  included  in  this 
investigation  because  it  was  more  relevant  to  target  prosecution  than  situational  awareness. 
Several  subjects  did  request  such  information,  which  indicates  its  importance  for  the  OOD,  but  it 
was  considered  outside  the  scope  of  the  present  research. 
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Design 


This  experiment  manipulated  experience  level  as  a  between  participant  factor.  For  the  concept 
rankings  measure,  operational  scenario  was  manipulated  via  factorial  combination  of 
presence/absence  of  enemy  and  shallow/deep  water. 

Procedure 

All  subjects  were  given  a  brief  overview  of  the  procedure  when  consent  forms  were 
administered,  and  the  research  was  conducted  in  compliance  with  all  applicable  federal 
regulations  governing  the  protection  of  human  subjects  in  research.  Prior  to  the  study, 
participants  completed  a  background  questionnaire  used  to  assess  their  level  of  experience  shown 
in  Appendix  B.  Next,  participants  judged  the  similarity  between  different  categories  of 
information  available  as  OOD.  This  judgment  of  similarity  is  a  reflection  of  the  conceptual 
organization  operators  employ.  Specifically,  subjects  participated  in  an  unconstrained  sorting 
task.  The  officers  received  a  set  of  20  index  cards  with  a  short  description  of  submarine 
information  printed  on  each,  and  they  arranged  these  cards  into  groups  according  to  similarity. 
The  definition  of  similarity  was  left  up  to  the  subject.  Cards  which  described  similar  categories 
were  placed  in  the  same  group,  and  any  card  which  described  a  unique  category  was  placed  by 
itself.  The  officers  could  create  as  many  or  as  few  groups  as  they  felt  appropriate. 

After  the  sorting  task,  the  officers  were  asked  to  rank  the  information  items  according  to 
importance.  They  ranked  the  items  four  times,  once  for  each  of  four  different  operational 
scenarios,  described  in  Appendix  C.  The  factors  that  varied  over  the  four  scenarios  were  the  type 
of  contact  (neutral  vs.  hostile)  and  the  type  of  environment  (deep  water  vs.  shallow  water).  The 
first  scenario  was  an  Antisubmarine  Warfare  (ASW)  patrol  in  a  shallow  water  environment  (i.e., 
littoral  sea;  "brown  water")  with  a  hostile  contact.  The  second  scenario  was  the  same  as  the  first, 
except  the  contact  was  neutral.  The  third  scenario  was  an  ASW  patrol  in  a  deep  water 
environment  (i.e.,  pelagic  ocean;  "blue  water")  with  a  hostile  contact.  Finally,  the  fourth 
scenario  was  the  same  as  the  third  except  the  contact  was  neutral.  These  four  scenarios  differ 
greatly  in  the  quality  and  quantity  of  information  available  to  the  OOD.  After  each  of  the  tasks, 
subjects  recorded  their  responses  on  answer  sheets  according  to  the  code  number  on  the  back  of 
each  stimulus  card. 


RESULTS 

For  the  card  sort  data,  in  order  to  assess  the  relation  among  concept  pairs,  each  possible  concept 
pair  (N  =  210)  was  coded  with  a  0  if  the  participant  did  not  group  them  in  the  same  category,  or  a 
1  if  they  were  grouped  in  the  same  category.  For  the  concept  rankings  data,  mean  concept 
importance  was  calculated  for  each  concept  across  participant  groups  and  scenarios. 

The  results  for  the  three  primary  groups  (SOBCpre,  SOBCpst,  SOAC)  are  presented,  and  where 
appropriate,  the  expert  composite  model  generated  from  the  PostDH  subjects  is  included. 
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Average  Correlation 


Mental  Model  Similarity  for  Card  Sort  Data  -  Overall  Correlations 

These  data  determine  the  degree  to  which  participants  may  differ  in  their  mental  models  of 
critical  OOD  concepts.  Using  the  concept  pair  coding  scheme  described  above,  correlations  with 
all  possible  participant  pairings  were  computed.  These  cross-correlations  were  run  across  and 
within  the  three  participant  groups  (SOAC,  SOBCpre,  SOBCpst).  Based  upon  this  matrix,  the 
data  had  six  conceptual  groups:  (1)  SOAC  correlated  with  SOAC;  (2)  SOAC  correlated  with 
SOBCpst;  (3)  SOAC  correlated  with  SOBCpre;  (4)  SOBCpre  correlated  with  SOBCpst;  (5) 
SOBCpst  correlated  with  SOBCpst;  and,  (6)  SOBCpre  correlated  with  SOBCpre.  These  data 
were  subjected  to  a  Univariate  ANOVA  and  resulted  in  a  significant  main  effect  for  group  type, 
F(5,  8250)  =  20.95,  p  <  .001.  For  ease  of  explication  these  data  are  presented  as  mean  within 
group  correlations  and  mean  across  group  correlations.  Specifically,  the  within  group 
correlations  correspond  to  "SOAC  correlated  with  SOAC,"  "SOBCpst  correlated  with 
SOBCpst,"  and  "SOBCpre  correlated  with  SOBCpre".  The  across  group  correlations  correspond 
to  "SOAC  correlated  with  SOBCpst,"  "SOAC  correlated  with  SOBCpre,"  and  "SOBCpre 
correlated  with  SOBCpst"  (Refer  to  Figure  1).  Unless  otherwise  specified,  the  reported  post-hoc 
analyses  are  all  significant  at  the  p  <  .05  level. 


with  with  with  with  with  with 

SOBCpst  SOBCpre  SOBCpst  SOBCpre  SOBCpst  SOAC 


Across  Group  Within  Group 

Correlations  Correlations 


Figure  1.  Mean  correlations  across  participant  pairings  resulting  from  card  sort  data. 
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When  considering  the  within  group  comparisons,  post-hoc  analyses  showed  that  the  degree  of 
mental  model  similarity  was  significantly  greater  for  the  more  experienced  (SOAC)  participants 
(M  =  .39).  The  degree  of  mental  model  similarity  was  significantly  lower  for  the  least 
experienced  (SOBCpre)  participants  ( M  =  .33)  with  mental  model  similarity  for  SOBCpst  in 
between,  and  significantly  different  from,  the  other  groups  ( M  =  .35).  When  considering  the 
across  group  comparisons,  post-hoc  analyses  showed  that  the  degree  of  across  group  mental 
model  similarity  was  greatest  for  SOAC  and  SOBCpst  (M  =  .36).  Correlations  between 
SOBCpre  and  SOBCpst  (M  =  .33)  and  between  SOAC  and  SOBCpre  (M  =  .33)  were  both 
significantly  lower  than  the  correlation  between  SOAC  and  SOBCpst. 

Years  of  Experience  and  Group  Correlations 

Next,  the  cross-correlations  based  upon  the  absolute  difference  in  Years  of  Experience  for  the 
participants  in  each  pair  were  parsed.  This  involved  computing  a  difference  score  in  the  "Years 
of  Experience"  for  all  participant  cross-correlations.  For  example,  if  one  participant  had  1 1  years 
of  experience,  and  the  other  had  2  years  of  experience,  their  difference  score  was  9  years  for  the 
correlation  on  which  they  were  compared.  Based  upon  this  difference  score  two  groups  of  cross¬ 
participant  correlations  were  created  for  more  detailed  analysis.  The  first  group  (low_diff) 
consisted  of  correlations  based  upon  participant  pairs  with  a  difference  score  less  than  2  years. 
The  second  group  (high_diff)  consisted  of  correlations  based  upon  participant  pairs  with  a 
difference  score  greater  than  8  years.  Groups  were  then  compared  in  order  to  determine  the 
degree  to  which  years  of  experience  may  alter  the  similarity  of  participants'  view  of  the  critical 
conceptual  relations.  For  illustrative  purposes,  Figure  2  shows  the  mean  correlations. 

When  considering  the  within  group  comparisons,  for  the  card  sort  data  from  novice  participants 
prior  to  training  (i.e.,  SOBCpre),  the  mean  correlation  for  the  low_diff  group  ( M  =  .34)  was 
numerically  greater  than  the  mean  correlation  for  the  high_diff  group  (M  =  .31).  This  difference 
revealed  a  trend  toward  significance,  F(l,  524)  =  2.73,  p  <  .10.  For  the  card  sort  data  from 
novice  participants  following  training  (i.e.,  SOBCpst),  the  mean  correlation  for  the  low_diff 
group  ( M  =  .38)  was  significantly  greater  than  the  mean  correlation  for  the  high_diff  group  ( M  = 
.33),  F(  1,  524)  =  10.20,  p  <  .01.  When  considering  the  card  sort  data  from  experienced 
participants  (i.e.,  SOAC),  the  mean  correlation  for  the  low_diff  group  (M  =  .41)  was  not 
significantly  different  (F  <  1)  from  the  mean  correlation  for  the  high_diff  group  ( M  =  .39). 

When  considering  the  across  group  comparisons  comparing  the  card  sort  data  from  novice 
participants  prior  to  and  following  training  (i.e.,  SOBCpre  and  SOBCpst),  the  mean  correlation 
for  the  low_diff  group  ( M  =  .35)  was  significantly  greater  than  the  mean  correlation  for  the 
high_diff  group  ( M  =  .31),  F(l,  1099)  =  17.89,  p  <  .001.  When  considering  the  across  group 
comparisons  comparing  the  novice  participants  prior  to  training  with  experienced  participants 
(i.e.,  SOBCpre  and  SOAC),  the  mean  correlation  for  the  low_diff  group  (M  =  .31)  was 
significantly  lower  than  the  mean  correlation  for  the  high_diff  group  ( M  =  .34),  F(l,  904)  =  4.36, 
p  <  .05.  Fast,  when  considering  the  across  group  comparisons  comparing  the  novice  participants 
following  training  with  experienced  participants  (i.e.,  SOBCpst  and  SOAC),  the  mean  correlation 
for  the  low_diff  group  ( M  =  .36)  was  not  significantly  different  from  the  mean  correlation  for  the 
high_diff  group  (M  =  .37),  (F  <  1). 
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Figure  2.  Mean  card  sort  correlations  for  all  groups  based  upon  years  experience  differential. 
Mental  Model  Similarity  for  Card  Sort  Data  -  Similarity  to  Expert  Composite  Model 

Additionally,  the  degree  to  which  these  participant  groups  generated  mental  models  similar  to  a 
composite  expert  model  based  on  the  PostDH  subjects  was  examined.  One  question  of  the 
present  research  was  whether  training  and/or  experience  influenced  structural  similarity  to  an 
expert  model,  and  it  was  hypothesized  that  the  differing  groups  would  show  differing  levels  of 
similarity  to  the  expert  model.  For  this  analysis  mean  correlations  with  the  expert  composite 
model  for  each  participant  group  were  computed.  An  ANOVA  revealed  a  main  effect  of  group  F 
(2,  126)  =  19.76,  p  <  .001.  Post-hoc  analyses  showed  that  the  SOAC  participants  (M  =  .43)  were 
significantly  higher  in  agreement  to  the  expert  model  than  were  the  SOBCpst  participants  (M  = 
.35),  who  were,  in  turn,  significantly  higher  than  the  SOBCpre  participants  (M  =  .28). 

Concept  Rankings  of  Importance  Compared  to  Expert  Composite  Model 

In  this  section  the  degree  to  which  submariners  view  concept  importance  differently  was 
analyzed.  This  analysis  has  both  theoretical  and  practical  value.  These  data  are  theoretically 
important  because,  from  the  mental  model  standpoint,  it  distinguishes  experts/novices  on  the 
basis  of  critical  misconceptions  in  item  importance.  These  data  are  practically  important  because 
these  misconceptions  can  be  considered  as  targets  for  training.  For  example,  this  allows  one  to 
identify  over-  and  under- statement  of  importance  for  trainees  at  different  levels.  This,  in  turn, 
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suggests  what  concepts  need  augmentation  in  the  training  (i.e.,  where  misconceptions  in  concept 
importance  lie  before  and  after  training). 

Next,  the  results  pertaining  to  the  degree  to  which  less  experienced  submariners  view  concept 
importance  differently  compared  to  the  expert  composite  model  is  discussed.  Cluster  analysis  of 
the  card  sort  data  for  the  expert  composite  group,  PostDH  subjects,  was  used  to  identify  clusters 
(reported  in  Shobe  &  Carr,  2004).  Refer  to  Shobe  &  Carr  (2004)  for  a  discussion  on  the 
differences  between  clusters  for  the  different  participant  groups.  The  clusters  identified  for  the 
expert  composite  group  are  shown  in  Table  3.  These  clusters  served  as  an  organizing  framework 
with  which  to  analyze  and  interpret  the  differences  in  importance  ratings  and  more  specifically 
target  problematic  concepts  within  the  overall  knowledge  structure.  For  the  participants'  priority 
ratings,  there  were  no  significant  differences  between  shallow  versus  deep  water  scenarios; 
therefore  only  effects  based  upon  experience  level  and  presence/absence  of  an  enemy  contact 
will  be  presented.  Only  the  overall  effects  for  the  rankings  data  and  the  comparisons  to  the 
expert  group  are  reported.  For  reference,  the  rankings  for  the  expert  group  are  shown  in 
Appendix  D.  For  illustrative  purposes,  these  concepts  are  graphed  according  to  a  difference 
score  calculated  using  the  mean  expert  score.  A  negative  score  means  participants  within  a  given 
group,  on  average,  viewed  this  concept  as  less  important  than  the  expert  composite  group.  For 
each  cluster  analyzed,  the  relevant  statistics,  tables,  and  figures  for  the  data  are  presented. 

Table  3.  Cluster  analysis  results  for  Post  DH  subjects. 


Group 

Item 

Own  Ship  Parameters/Contact  Management 

Own  ship  data 

Electronic  Support  Measures  (ESM)  contact  data 

Visual  contact  data 

Geosit/ops  summary 

Trial  own  ship 

Non-target  fire  control  solution 

Target  fire  control  solution 

Ship  System  Status 

Propulsion  Plant  Lineup 

Ship's  atmosphere 

Potable  water  status 

Sanitary  tank  status 

Equipment  Lineup/S earch  Data 

Counterdetectability  data 

Sonar  search  plan 

Active  sonar  lineup 

Sonar  lineup 

Ocean  environment  and  navigation  data 

Sonar  Tactical  Displays 

Performance  Monitoring/Fault  Location  (PMFL)  data 
Sonar  tracker/cursor  audio 

Sonar  detection  displays 

Sonar  class  displays 
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Own  Ship  Parameters  and  Contact  Management  Cluster 

Table  4  shows  the  concepts  for  the  "Own  Ship  Parameters  and  Contact  Management"  cluster 
identified  for  the  expert  group.  The  mean  concept  importance  ratings  for  the  four  groups 
(SOBCpre,  SOBCpst,  SOAC,  PostDH)  were  subjected  to  a  Multivariate  ANOVA.  Group  type 
was  the  independent  factor  and  the  seven  concepts  identified  in  Table  4  were  the  dependent 
variables.  There  was  a  significant  multivariate  main  effect  for  group  type,  Fill ,  1560)  =  6.51  ,p 
<  .001.  For  ease  of  explication,  Appendix  E  presents  the  between  subjects  effects  for  this 
analysis.  Similarly,  Figure  3  illustrates  the  pattern  of  differences  for  the  effect  of  group  type  on 
mean  concept  rankings  for  the  Own  Ship  Parameters  and  Contact  Management  cluster. 
Significant  differences  between  the  expert  group  and  the  participant  groups  are  noted  in  the 
footnotes  in  Appendix  E. 

Table  4.  Concept  names  and  explanations  for  the  Own  Ship  Parameters  and  Contact 
Management  cluster. 


Card 

Concept 

Description 

1 

ESM  contact  data 

Bearing  and  classification  of  radar  and  communications 
systems  data  provided  by  the  Electronic  Support  Measures 
system 

2 

Own  ship  data 

Own  ship's  course,  speed,  depth,  etc. 

8 

Visucd  contact  data 
( from  photonics  mast) 

Visual  information  on  surface  contacts  including  night 
vision,  infrared,  video,  laser  ranging,  etc. 

14 

Geosit/ops  summary 

Computer  generated  geographical  picture  of  contacts  and 
own  ship,  in  either  true  or  relative  bearing  orientation,  with 
classification  information  where  possible 

15 

Tried  own  ship 

Closest  Point  of  Approach  (CPA)  solutions,  trial  maneuvers, 
etc.,  i.e.,  data  which  aid  in  assessment  of  the  present  and 
future  tactical  situation 

16 

Non-target  fire  control 
solution 

Rapid  passive  localization  by  wide  aperture  array  (WAA), 
KAST  ranging,  multipath  ranging,  D/E  ranging,  hyperbolic 
ranging;  Bearing,  range,  course,  speed,  depth  if  submerged 
contact,  for  secondary  contacts 

19 

Target  fire  control 
solution 

Rapid  passive  localization  by  wide  aperture  array  (WAA), 
KAST  ranging,  multipath  ranging,  D/E  ranging,  hyperbolic 
ranging;  Bearing,  range,  course,  speed,  depth  if  submerged 
contact,  for  target  of  interest 
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Figure  3.  Ranking  difference  scores  for  Own  Ship  Parameters  and  Contact  Management  Cluster. 


Ship  System  Status  Cluster 

Table  5  lists  and  defines  the  concepts  making  up  the  "Ship  System  Status"  cluster.  The  mean 
concept  importance  ratings  for  the  four  groups  (SOAC,  SOBCpre,  SOBCpst,  PostDH)  was 
subjected  to  a  Multivariate  ANOVA.  Group  type  was  the  independent  factor  and  the  four 
concepts  identified  in  Table  5  were  the  dependent  variables.  There  was  a  significant  multivariate 
main  effect  for  group  type,  F(12,  1569)  =  5.14,  p  <  .001.  For  ease  of  explication,  Appendix  F 
presents  the  between  subjects  effects  for  this  analysis.  Similarly,  Figure  4  illustrates  the  pattern 
of  differences  for  the  effect  of  group  type  on  mean  concept  rankings  for  the  Sonar  Tactical 
Displays  cluster.  Significant  differences  between  the  expert  group  and  the  participant  groups  are 
noted  in  the  footnotes  for  Appendix  F. 
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Table  5.  Concept  names  and  explanations  for  the  Ship  System  Status  cluster. 


Card 

Concept 

Description 

6 

Propulsion  Plant 
Lineup 

Operating  condition  of  the  reactor  and  propulsion  equipment 
including  reactor  coolant  pump  speed  (fast,  slow,  reduced 
frequency  operations  (RFO);  electric  plant  lineup  (full 
power,  half  power,  other);  any  limits  on  operations  due  to 
casualties  or  maintenance  in  the  engineering  spaces 

9 

Potable  water  status 

Amount  of  water  in  potable  water  tanks  and  tank  on  service 

11 

Ship's  atmosphere 

Status  of  atmosphere  inside  the  submarine  (oxygen 
percentage,  C02  concentration  percentage,  CO/FF 
concentrations);  status  of  oxygen  bleed  from  02  banks;  status 
of  oxygen  generator  or  02  candle  furnace;  status  of  C02 
scrubbers  and  CO/H2  burner 

13 

Sanitary  tank  status 

Tank  levels  in  ship's  sanitary  tanks  and  status  of  pumping  or 
blowing  tanks 
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Figure  4.  Ranking  difference  scores  for  Ship  System  Status  Cluster. 
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Equipment  Lineup/Search  Data  Cluster 

Table  6  lists  and  defines  the  concepts  making  up  the  "Equipment  Lineup/Search  Data"  cluster. 
The  mean  concept  importance  ratings  for  the  four  groups  (SOAC,  SOBCpre,  SOBCpst,  PostDH) 
were  subjected  to  a  Multivariate  ANOVA.  Group  type  was  the  independent  factor  and  the  five 
concepts  identified  in  Table  6  were  the  dependent  variables.  There  was  a  significant  multivariate 
main  effect  for  group  type,  F(15,  1563)  =  1 1.76,  p  <  .001.  For  ease  of  explication,  Appendix  G 
presents  the  between  subjects  effects  for  this  analysis.  Similarly,  Figure  5  illustrates  the  pattern 
of  differences  for  the  effect  of  group  type  on  mean  concept  rankings  for  the  Sonar  Tactical 
Displays  cluster.  Significant  differences  between  the  Expert  group  and  the  participant  groups  are 
noted  in  the  footnotes  for  Appendix  G. 


Table  6.  Concept  names  and  explanations  for  the  Equipment  Lineup/Search  Data  cluster. 


Card 

Concept 

Description 

4 

Ocean  environment 
and  navigation  data 

Sea  state;  bottom  depth;  type,  and  features;  background  noise; 
sound  velocity  profile;  propagation  loss,  i.e.,  computer 
generated  data  used  to  position  own  ship  to  maximize  sonar 
detection  range  (Le) 

5 

Counter- 
detectability  data 

Own  ship  electromagnetic  signature,  radiated  noise,  surface 
visibility /wake,  contaminants,  etc. 

7 

Sonar  search  plan 

Physical  lineup  of  sonar  system,  including  operator  displays 
and  processing  used;  search  track  to  be  followed  by  ship  to 
locate  target  of  interest;  depth  to  search  at  based  on  ocean 
bathymetric  data 

10 

Active  sonar  lineup 

Power,  pulse  type,  range  gate,  etc.,  i.e.,  parameters  indicating 
active  sonar  transmission  status 

18 

Sonar  lineup 

Narrow  band  and  broadband  detection  system  lineup,  filter  and 
alarm  settings,  other  operator  set  parameters 

19 
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Figure  5.  Ranking  difference  scores  for  Equipment  Lineup/Search  Data  Cluster. 


Sonar  Tactical  Displays  Cluster 

Table  7  lists  and  defines  the  concepts  making  up  the  "Sonar  Tactical  Displays"  cluster  for  the 
expert  group.  The  mean  concept  importance  ratings  for  the  four  groups  (SOAC,  SOBCpre, 
SOBCpst,  PostDH)  were  subjected  to  a  Multivariate  ANOVA.  Group  type  was  the  independent 
factor  and  the  four  concepts  identified  in  Table  7  were  the  dependent  variables.  There  was  a 
significant  multivariate  main  effect  for  group  type,  FT  12,  1569)  =  12.06,  p  <  .001.  For  ease  of 
explication,  Appendix  H  presents  the  between  subjects  effects  for  this  analysis.  Similarly,  Figure 
6  illustrates  the  pattern  of  differences  for  the  effect  of  group  type  on  mean  concept  rankings  for 
the  Sonar  Tactical  Displays  cluster.  Significant  differences  between  the  expert  group  and  the 
participant  groups  are  noted  in  the  footnotes  of  Appendix  H. 
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Table  7.  Concept  names  and  explanations  for  the  Sonar  Tactical  Displays  cluster. 


Card 

Concept 

Description 

3 

PMFL  data 

Data  on  fault  location  and  performance  monitoring  of  various  sonar 
components,  i.e.,  data  required  to  maintain  electrical  information 
systems 

12 

Sonar 

tracker/cursor 

audio 

Auditory  presentation  of  minimally  processed  sonar  signals  as 
picked  up  by  the  various  arrays 

17 

Sonar  detection 
displays 

Visual  displays  of  sonar  detections  as  presented  to  sonar  operators 

20 

Sonar  class 
displays 

Sonar  signal  interpretation  aids  such  as  signature  assemblies, 
lofargrams,  etc.,  i.e.,  data  used  to  aid  in  the  classification  of  signals 

9 

7  QSOBCpre 

■  SOBCpost 
□  SOAC 


5 


-7  PMFL  Data  Sonar  Tracker/  Sonar  Detection  Sonar  Class 

Cursor  Audio  Displays  Displays 


-9 

Concepts  for  Sonar  Tactical  Displays 


Figure  6.  Ranking  difference  scores  for  Sonar  Tactical  Displays  Cluster. 

The  previous  analysis  compared  the  ranking  scores  for  the  various  subject  groups  to  the  expert 
composite  model.  This  analysis  was  done  for  the  concepts  within  each  of  the  four  clusters 
identified  in  the  PostDH  cluster  analysis.  For  the  Own  Ship  Parameters  and  Contact 
Management  cluster,  the  less  experienced  personnel  consistently  ranked  ESM  contact  data, 
visual  contact  data,  and  target  fire  control  solution  as  more  important  than  the  experts,  while  own 
ship  data,  geosit/ops  summary,  and  nontarget  fire  control  solution  were  ranked  as  less  important 
than  experts.  Visual  contact  data  demonstrated  the  largest  discrepancy  between  groups.  For  the 
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ship  system  status  cluster,  SOAC  subjects  consistently  ranked  the  items  as  less  important  than 
experts  (except  for  sanitary  tank  status),  with  minor  differences  for  the  SOBCpre  and  SOBCpst 
subjects.  The  largest  difference  in  the  equipment  lineup/search  data  cluster  were  the  rankings  for 
sonar  lineup  which  SOAC  subjects  ranked  as  less  important  than  experts.  The  other  differences 
between  groups  and  items  were  minimal.  Finally,  for  the  sonar  tactical  displays  cluster,  the  less 
experienced  personnel  ranked  sonar  detection  displays  and  sonar  class  displays  as  less  important 
than  experts.  These  results  pinpoint  concept  areas  that  should  be  the  focus  of  training  for  the 
less  experienced  submarine  officers. 


DISCUSSION 

This  report,  in  conjunction  with  Shobe  &  Carr  (2004),  demonstrates  significant  differences  in  the 
way  submarine  officers  conceptually  organize  and  prioritize  concepts  available  to  the  Officer  of 
the  Deck.  Even  though  each  subject  group  organized  the  concepts  along  two  dimensions  (Shobe 
&  Carr,  2004),  the  organization  within  these  dimensions  varied  as  a  result  of  experience. 
Moreover,  as  the  personnel  gained  more  experience,  they  required  few  clusters  to  classify  the 
information  items.  Within  these  clusters,  differences  in  concept  importance  were  identified 
compared  to  an  expert  composite  model. 

In  support  of  shared  mental  model  theory  and  consistent  with  other  studies  of  knowledge 
organization  and  the  mental  models  of  submariners  (e.g.,  Smith-Jentsch  et  al.,  2001),  this 
investigation  found  that  more  experienced  personnel  view  conceptual  linkages  more  similarly 
and  showed  higher  agreement  with  an  expert  model  (Fiore  et  al.,  2000).  Conversely,  the  least 
experienced  personnel  see  conceptual  linkages  less  similarly,  but,  following  training,  these 
participants  were  found  to  view  the  conceptual  linkages  more  similarly  to  the  experienced 
personnel.  From  the  standpoint  of  understanding  the  importance  of  particular  concepts, 
significant  differences  between  experienced  and  novice  personnel  were  identified  prior  to,  and 
following,  training.  These  data  highlight  how  participants  vary  in  their  understanding  of  the 
importance  of  these  concepts. 

Finking  the  conceptual  groupings  with  the  priorities  data  can  better  illuminate  knowledge 
organization  and  the  differing  ways  that  knowledge  may  be  tapped  in  operational  environments 
(cf.  Schvaneveldt  et  al.,  2001).  From  the  standpoint  of  training,  these  data  suggest  that  the 
conceptual  grouping  garnered  from  the  expert  group  may  aid  in  conveying  important  clusters  of 
information  as  well  as  how  priorities  vary  within  these  clusters.  These  data  are  theoretically 
important  because,  from  the  mental  model  standpoint,  it  distinguishes  more  and  less  experienced 
personnel  on  the  basis  of  critical  misconceptions  in  item  importance.  These  data  are  practically 
important  because  these  misconceptions  can  be  considered  as  targets  for  training.  For  example, 
this  allows  one  to  identify  over-  and  under- statement  of  importance  for  trainees  at  different 
levels.  This,  in  turn,  suggests  what  concepts  need  augmentation  in  the  training  (i.e.,  where 
misconceptions  in  concept  importance  lie  before  and  after  training)  and  may  help  with  redesign 
of  systems  based  upon  the  importance  of  data. 
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APPENDIX  A  Information  Categories  Available  to  the  OOD  at  CONN 


1.  ESM  contact  data 

Bearing  and  classification  of  radar  and  communications  systems  data 
provided  by  the  Electronic  Support  Measures  system 

2.  Own  ship  data 

Own  ship's  course,  speed,  depth,  etc. 

3.  PMFL  data 

Data  on  fault  location  and  performance  monitoring  of  various  sonar 
components,  i.e.,  data  required  to  maintain  electrical  information  systems 

4.  Ocean  environment 
and  navigation  data 

Sea  state;  bottom  depth;  type,  and  features;  background  noise;  sound  velocity 
profile;  propagation  loss,  i.e.,  computer  generated  data  used  to  position  own 
ship  to  maximize  sonar  detection  range  (Le) 

5.  Counterdetectability 
data 

Own  ship  electromagnetic  signature,  radiated  noise,  surface  visibility/wake, 
contaminants,  etc. 

6.  Propulsion  Plant 
Lineup 

Operating  condition  of  the  reactor  and  propulsion  equipment  including  reactor 
coolant  pump  speed  (fast,  slow,  RFO);  electric  plant  lineup  (full  power,  half 
power,  other);  any  limits  on  operations  due  to  casualties  or  maintenance  in  the 
engineering  spaces 

7.  Sonar  search  plan 

Physical  lineup  of  sonar  system,  including  operator  displays  and  processing 
used;  search  track  to  be  followed  by  ship  to  locate  target  of  interest;  depth  to 
search  at  based  on  ocean  bathymetric  data 

8.  Visual  contact  data 
(from  photonics  mast) 

Visual  information  on  surface  contacts  including  night  vision,  infrared,  video, 
laser  ranging,  etc.  (Note:  only  available  at  PD) 

9.  Potable  water  status 

Amount  of  water  in  potable  water  tanks  and  tank  on  service 

10.  Active  sonar  lineup 

Power,  pulse  type,  range  gate,  etc.,  i.e.,  parameters  indicating  active  sonar 
transmission  status 

11.  Ship’s  atmosphere 

Status  of  atmosphere  inside  the  submarine  (oxygen  percentage,  C02 
concentration  percentage,  CO/PE  concentrations);  status  of  oxygen  bleed  from 
02  banks;  status  of  oxygen  generator  or  02  candle  furnace;  status  of  C02 
scrubbers  and  CO/H2  burner 

12.  Sonar  tracker/cursor 
audio 

Auditory  presentation  of  minimally  processed  sonar  signals  as  picked  up  by 
the  various  arrays 

13.  Sanitary  tank  status 

Tank  levels  in  ship's  sanitary  tanks  and  status  of  pumping  or  blowing  tanks 

14.  Geosit/ops  summary 

Computer  generated  geographical  picture  of  all  contacts  and  own  ship,  in 
either  true  or  relative  bearing  orientation,  with  classification  information 
where  possible 

15.  Trial  own  ship 

CPA  solutions,  trial  maneuvers,  etc.,  i.e.,  data  which  aid  in  assessment  of  the 
present  and  future  tactical  situation 

16.  Non-target  fire 
control  solution 

Rapid  passive  localization  by  wide  aperture  array  (WAA),  KAST  ranging, 
multipath  ranging,  D/E  ranging,  hyperbolic  ranging;  Bearing,  range,  course, 
speed,  depth  if  submerged  contact,  for  secondary  contacts 

17.  Sonar  detection 
displays 

Visual  displays  of  sonar  detections  as  presented  to  sonar  operators 

18.  Sonar  lineup 

Narrow  band  and  broadband  detection  system  lineup,  filter  and  alarm  settings, 
other  operator  set  parameters 

19.  Target  fire  control 
solution 

Rapid  passive  localization  by  wide  aperture  array  (WAA),  KAST  ranging, 
multipath  ranging,  D/E  ranging,  hyperbolic  ranging;  Bearing,  range,  course, 
speed,  depth  if  submerged  contact,  for  target  of  interest 

20.  Sonar  class  displays 

Sonar  signal  interpretation  aids  such  as  signature  assemblies,  lofargrams,  etc. 
i.e.,  data  used  to  aid  in  the  classification  of  signals 
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APPENDIX  B  Background  Survey 

Please  provide  responses  to  the  following  to  help  us  evaluate  the  results  of  this  experiment. 

1 .  Present  rank  _ 

2.  Years  in  rank _ 

3.  Age _ 

4.  Years  in  service  _ _ 

5.  Years  Sea  Duty  . _ 

6.  Sea  Billets _ 


7.  Present  billet _ 

8.  Training  (e.g.,  SOBC,  SOAC,  JO  courses,  PXO,  PCO) 
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APPENDIX  C  Scenarios  Used  for  Prioritization  Task 


INSTRUCTIONS  FOR  RANKING  TASK  1 

We  would  now  like  you  to  reassemble  the  cards  into  a  single  deck.  Then,  rank  the 
information  on  the  cards  according  to  importance  for  display  at  CONN. 

For  context,  the  operational  environment  should  be  assumed  to  be  a  SSN  in  the  following 
scenario: 

Mission:  (littoral,  enemy  contacts) 

The  world  situation  is  that  a  war  has  broken  out  between  Russia  and  Ukraine  over  control  of  the 
formerly  Soviet  Navy.  Both  nations  are  trying  to  get  the  U.S.  involved  and  have  been  attacking 
U.S.  shipping  with  submarines  and  then  blaming  each  other.  You  are  patrolling  in  the  eastern 
Mediterranean  Sea.  Your  mission  is  to  protect  U.S.  shipping  lanes.  Your  orders  are  to  search 
and  destroy  any  enemy  submarines  in  the  vicinity  of  U.S.  shipping.  There  are  several  surface 
contacts  (i.e.,  merchants,  fishing  trawlers),  and  no  subsurface  contacts.  The  broad  band  range  of 
the  day  for  a  subsurface  contact  is  3-6  Kyd.  Intelligence  has  indicated  a  Kilo-type  Formerly 
Soviet  Submarine  is  patrolling  in  the  area.  There  are  no  friendly  submarines  in  your  patrol  area. 
The  month  is  April  and  the  sea  state  is  2. 

OS  Course:  120  Speed:  5  Operating  Depth:  250-400ft 


INSTRUCTIONS  FOR  RANKING  TASK  2 

Again,  we  would  like  you  to  reassemble  the  cards  into  a  single  deck.  Then,  rank  the 
information  on  the  cards  according  to  importance  for  display  at  CONN. 

For  context  and  unlike  Ranking  Task  1,  the  operational  environment  should  be  assumed 
to  be  a  SSN  in  the  following  scenario: 

Mission:  (littoral,  no  enemy  contacts) 

The  world  situation  is  that  it  is  peacetime  and  there  are  no  conflicts  brewing.  You  are  on  a 
training  exercise  in  the  eastern  Mediterranean  Sea.  There  are  several  surface  contacts  (i.e., 
merchants,  fishing  trawlers)  and  there  is  a  friendly  submarine  in  your  patrol  area,  a  Trafalgar 
class  British  sub.  The  broad  band  range  of  the  day  for  a  subsurface  contact  is  3-6  Kyd.  The 
month  is  April  and  the  sea  state  is  2. 

OS  Course:  120  Speed:  5  Operating  Depth:  250-400ft 
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INSTRUCTIONS  FOR  RANKING  TASK  3 


We  would  now  like  you  to  reassemble  the  cards  into  a  single  deck.  Then,  rank  the 
information  on  the  cards  according  to  importance  for  display  at  CONN. 

Unlike  the  previous  scenarios,  the  operational  environment  should  be  assumed  to  be  a 
SSN  in  the  following  scenario: 

Mission:  ( deep  water,  enemy  contacts) 

The  world  situation  is  that  a  war  has  broken  out  between  Russia  and  Ukraine  over  control  of  the 
formerly  Soviet  Navy.  Both  nations  are  trying  to  get  the  U.S.  involved  and  have  been  attacking 
U.S.  shipping  with  submarines  and  then  blaming  each  other.  You  are  underway  from  the  East 
Coast  and  are  cruising  along  to  the  eastern  Mediterranean  Sea.  Once  you  arrive,  your  mission  is 
to  protect  U.S.  shipping  lanes.  Your  orders  are  to  search  and  destroy  any  enemy  submarines  in 
the  vicinity  of  U.S.  shipping.  While  you  are  underway,  in  the  middle  of  the  Atlantic  Ocean, 
intelligence  has  indicated  a  Kilo-type  Formerly  Soviet  Submarine  is  in  the  area.  The  month  is 
April  and  the  sea  state  is  2. 

OS  Course:  120  Speed:  20  Operating  Depth:  500 


INSTRUCTIONS  FOR  RANKING  TASK  4 

We  would  now  like  you  to  reassemble  the  cards  into  a  single  deck.  Then,  rank  the 
information  on  the  cards  according  to  importance  for  display  at  CONN. 

Unlike  the  previous  scenarios,  the  operational  environment  should  be  assumed  to  be  a 
SSN  in  the  following  scenario: 

Mission:  ( deep  water,  no  enemy  contacts) 

The  world  situation  is  that  it  is  peacetime  and  there  are  no  conflicts  brewing.  You  are  underway 
in  the  Atlantic  Ocean  to  a  training  exercise  in  the  eastern  Mediterranean  Sea.  There  are  a  few 
commercial  surface  contacts  and  there  is  a  friendly  submarine  in  the  area,  a  Trafalgar  class 
British  sub.  The  broad  band  range  of  the  day  for  a  subsurface  contact  is  3-6  Kyd.  The  month  is 
April  and  the  sea  state  is  2. 

OS  Course:  120  Speed:  20  Operating  Depth:  500 
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APPENDIX  D  Ranking  Data  for  Post  DH  Participants 


Rank 

Deep/Neutral 

Deep/Hostile 

Shallow/Neutral 

Shallow/Hostile 

1 

Sonar  Detection  Displays 

Sonar  Detection  Displays 

Sonar  Detection  Displays 

Sonar  Detection  Displays 

2 

Target  Fire  Control  Solution 

Own  Ship  Data 

Own  Ship  Data 

Own  Ship  Data 

3 

Own  Ship  Data 

Geosit/Ops  Summary 

Geosit/Ops  Summary 

Geosit/Ops  Summary 

4 

Geosit/Ops  Summary 

NonTarget  Lire  Control  Solution 

NonTarget  Lire  Con  tied  Solution 

Sonar-  Class  Displays 

5 

NonTarget  Fire  Control  Solution 

Sonar  Class  Displays 

Target  Lire  Control  Solution 

NonTarget  Lire  Control  Solution 

6 

Sonar  Class  Displays 

Counterdetectability  Data 

Counterdetectability  Data 

Ocean  Environment 

7 

Trial  Own  Ship 

Ocean  Environment 

Trial  Own  Ship 

Trial  Own  Ship 

8 

Ocean  Environment 

Sonar  Tracker/Cursor  Audio 

Sonar  Class  Displays 

Counterdetectability  Data 

9 

Sonar-  Search  Plan 

Trial  Own  Ship 

Ocean  Environment 

Propulsion  Plant  Lineup 

10 

Sonar-  Lineup 

Propulsion  Plant  Lineup 

Propulsion  Plant  Lineup 

Sonar  Search  Plan 

11 

Counterdetectability  Data 

Sonar-  Lineup 

Sonar  Lineup 

Sonar-  Lineup 

12 

ESM  Contact  Data 

Target  Lire  Control  Solution 

Sonar-  Search  Plan 

Sonar  Tracker/Cursor  Audio 

13 

Sonar-  Tracker/Cursor  Audio 

Ship's  Atmosphere 

Sonar  Tracker/Cursor  Audio 

Target  Lire  Control  Solution 

14 

Active  Sonar  Lineup 

Sonar  Search  Plan 

Active  Sonar  Lineup 

Ship's  Atmosphere 

15 

Propulsion  Plant  Lineup 

Visual  Contact  Data 

PMLL  Data 

Potable  Water  Status 

16 

Ship's  Atmosphere 

Potable  Water  Status 

Ship's  Atmosphere 

PMLL  Data 

17 

PMLL  Data 

Sanitary  Tank  Status 

ESM  Contact  Data 

ESM  Contact  Data 

18 

Visual  Contact  Data 

ESM  Contact  Data 

Potable  Water  Status 

Visual  Contact  Data 

19 

Potable  Water  Status 

PMLL  Data 

Visual  Contact  Data 

Active  Sonar  Lineup 

20 

Sanitary  Tank  Status 

Active  Sonar  Lineup 

Sanitary  Tank  Status 

Sanitary  Tank  Status 
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APPENDIX  E  Results  from  Multivariate  Analysis  for  Own  Ship  Parameters  and  Contact 
Management  Cluster 


Source 

DV 

Type  III  SS 

df 

Mean  Square 

F 

Sig. 

Participant  Group 

C01a 

803.098 

3 

267.699 

10.359 

.000 

C02e 

61.430 

3 

20.477 

1.274 

.283 

C08a 

1813.450 

3 

604.483 

18.289 

.000 

C14c 

466.602 

3 

155.534 

9.513 

.000 

C15 e 

302.838 

3 

100.946 

5.417 

.001 

C16b 

410.182 

3 

136.727 

7.004 

.000 

C19d 

368.573 

3 

122.858 

5.398 

.001 

Error 

cot 

13540.902 

524 

25.841 

C08 

17319.459 

524 

33.052 

C14 

8566.914 

524 

16.349 

C15 

9764.677 

524 

18.635 

C16 

10228.477 

524 

19.520 

C19 

11925.379 

524 

22.758 

aSOBCpre,  SOBCpst,  and  SOAC  significantly  different  from  Expert  Group 
b  SOBCpre  and  SOBCpst  significantly  different  from  Expert  Group 
c  SOBCpst  significantly  different  from  Expert  Group 
d  SOAC  significantly  different  from  Expert  Group 
e  None  significantly  different  from  Expert  Group 
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APPENDIX  F  Results  from  Multivariate  Analysis  for  Ship  System  Status  Cluster 


Source 

DV 

Type  III 

SS 

df 

Mean 

Square 

F 

Sig. 

Participant 

C06a 

773.77 

3 

257.92 

10.23 

0.00 

Group 

C09a 

144.73 

3 

48.24 

3.51 

0.02 

Clla 

50.76 

3 

16.92 

1.01 

0.39 

C13 a 

208.19 

3 

69.40 

5.60 

0.00 

Error 

C06 

13207.86 

524 

25.21 

C09 

7213.45 

524 

13.77 

Cll 

8765.30 

524 

16.73 

C13 

6496.58 

524 

12.40 

a  None  significantly  different  from  Expert  Group 
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APPENDIX  G  Results  from  Multivariate  Analysis  for  Equipment  Lineup/Search  Data 
Cluster 


Source 

DV 

Type  III 

SS 

df 

Mean 

Square 

F 

Sig. 

Participant 

C04a 

427.62 

3 

142.54 

7.38 

0.00 

Group 

C05 a 

926.12 

3 

308.71 

17.09 

0.00 

C07a 

485.87 

3 

161.96 

7.29 

0.00 

C10b 

200.92 

3 

66.97 

4.40 

0.01 

C18c 

1258.96 

3 

419.65 

21.98 

0.00 

Error 

C04 

10095.32 

523 

19.30 

C05 

9447.38 

523 

18.06 

C07 

11617.91 

523 

22.21 

CIO 

7954.42 

523 

15.21 

C18 

9987.62 

523 

19.10 

aNone  significantly  different  from  Expert  Group 
b  SOBCpre  significantly  different  from  Expert  Group 
c  SOAC  significantly  different  from  Expert  Group 
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APPENDIX  H  Results  from  Multivariate  Analysis  for  Sonar  Tactical  Displays  Cluster 


Source 

DV 

Type  III  SS 

df 

Mean  Square 

F 

Sig. 

Participant  Group 

C03a 

451.424 

3 

150.475 

12.091 

.000 

C12a 

97.034 

3 

32.345 

1.634 

.180 

C17b 

2464.951 

3 

821.650 

44.870 

.000 

C20c 

421.197 

3 

140.399 

6.511 

.000 

Error 

C03 

6521.256 

524 

12.445 

C12 

10370.959 

524 

19.792 

C17 

9595.292 

524 

18.312 

C20 

11298.864 

524 

21.563 

aNone  significantly  different  from  Expert  Group 
b  SOBCpre  and  SOBCpst  significantly  different  from  Expert  Group 
c  SOBCpre,  SOBCpst,  and  SOAC  significantly  different  from  Expert  Group 
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14.  Abstract  (cont) 

Eighty-three  Naval  Officers  performed  a  card  sorting  and  ranking  task  to  indicate  similarity  and 
relative  importance  among  20  categories  of  information  available  on  submarines  in  different 
operational  environments.  These  submarine  concepts  consisted  of  types  of  information  the 
submarine  OOD  would  encounter  while  on  watch,  with  the  exception  of  tactics  information. 
Participants  were  instructed  to  sort  the  cards  into  piles  according  to  similarity  and  were  not 
constrained  by  the  number  of  piles  they  could  create.  Following  the  sorting  task  participants 
ranked  the  cards  according  to  relative  importance  for  four  scenarios.  They  were  instructed  to 
create  a  single  pile  of  cards  ordered  from  the  most  important  piece  of  submarine  information  to 
the  least  important.  Data  were  collected  from  Submarine  Officer  Basic  Course  (SOBC)  students 
pre  and  post  course,  Submarine  Officer  Advanced  Course  (SOAC)  students,  and  Post 
Department  Head  submariners  (PostDH). 

Correlations  between  and  among  participant  groups  (SOBCpre,  SOBCpst,  SOAC,  PostDH) 
revealed  differences  in  cognitive  organization  of  the  submarine  OOD  concepts.  Within  group 
comparisons  showed  that  the  degree  of  mental  model  similarity  was  significantly  greater  for 
more  experienced  participants.  When  considering  across  group  comparisons,  analyses  showed 
that  the  degree  of  across  group  mental  model  similarity  was  greatest  for  SOAC  and  SOBCpst. 
Moreover,  SOAC  students  were  in  higher  agreement  with  the  expert  composite  model  generated 
from  Post  DH  submariners,  and  concept  rankings  varied  as  a  function  of  experience. 

In  support  of  shared  mental  model  theory  and  consistent  with  other  studies  of  knowledge 
organization  and  the  mental  models  of  submariners,  this  investigation  found  that  more 
experienced  personnel  view  conceptual  linkages  more  similarly  and  showed  higher  agreement 
with  an  expert  model  (Fiore,  Fowlkes,  Martin-Milham,  &  Oser,  2000).  Conversely,  the  least 
experienced  personnel  see  conceptual  linkages  less  similarly,  but,  following  training,  these 
participants  were  found  to  view  the  conceptual  linkages  more  similarly  to  the  experienced 
personnel.  From  the  standpoint  of  understanding  the  importance  of  particular  concepts, 
significant  differences  between  experienced  and  novice  personnel  were  identified  prior  to,  and 
following,  training.  These  data  highlight  how  participants  vary  in  their  understanding  of  the 
importance  of  these  concepts 
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